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the major problems encountered by the administrators; and second, 
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sketch out a portrait of the situation. The evaluation results and 
findings were transformed into specific recommendations to improve 
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information as to the evaluation process and product; (3) involve a 
specialist to corroborate data; and (4) use constant feedback as an 
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Abstract 

The responsive m evaluation has been seen as an alternative ti 

formal evaluation .^1els. The model has been used for the evaluation 
of a program in a Quebec French university. The study was focusing 
on two main objectives: first, the local utility of the program being 
assessed and, the major probl ems encountered by the admi nistrators 
and, second, identifying the advantages and the limits of the model, 
comparing them to those reported in ' ne literature. As far as the 
first objective is concerned, the rid , detailed and pertinent infor- 
mation enabled the evaluator to s* 3tch out a portrait of the 
situation. The evaluation results and findinqs were transformed into 
specific recommandations apt to improve the program. As for the 
second objective, several advantages and limits cited in the 
literature were confirmed whereas some n?w ones were found. Specific 
recommandations about the use of the responsive model re presented 
as concluding remarks. 
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Introduction 

For the last decade, criticisms have been adressed to formal 
model s of programs 1 eval uation (Guba , 1 969 ; Stake, 1976; Guba and 
Lincoln, 1981; Borich and Jemelka, 1982). Among the most significant 
ones, we find: methodological inadequacies of the models; their lack 
of comprehension of the social and political contexts; the biases of 
the formal approach mor preoccupied by influential audiences than by 
the concerned ones; the lack of realism of the criteria used to 
measure the success of a program; t h e irrelevance of the approach 
sinca it is concerned with questions of no interest to the different 
audiences; finally the uselesness of the models is stressed since 
more often than not the situation remains unchanges. Weiss stressed 
the fol 1 owi ng: 

1. the emphasis is placed on the concerns of those who have the 
power rather than on those involved in th?. program (Berk and Rossi, 
1976; Parlett and Hamilton, 1976; Patton, 1978; Stake, 1978; Cochran, 
1980; Coleman, 1980; Houce, 1980; Datta, 1981); 

2. the criteria used to measure the success o* a program lack 
realism (Caro, 1971 r Schwartz, 1980); 

3. the results are often useless since they do not influence the 
decision making (Weiss, 1972; Scott and Shore, 1979; Rutman, 1980); 

4. the methods used in research are more often than not inadequate 
in evaluation (Cochran, 1980). 
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Finally, Gold (1981) argues that in the past and in order to 
avoid confrontations with their collegues, evaluators were more 
concerned about methodology tnan by the beneficiaries preoccupations. 

An Al ternat ive : the naturalistic evaluation 
The naturalistic approach to evaluation has been presented as an 
alternative to the traditionnal models of programs' evaluation 
(Levine, 1974; Stake, 1976; Parlett and Hamilton, 1976; Guba et 
Lincoln, 1981). This approach presents the following cha^ac 
teristics: 

- it is centered on che activities of a program rather than on its 
i ntents; 

- it favors any instruments that wil 1 al 1 ow the gatheri ng of the 
needed i nf orma + i on; 

- it is responsive to the values of the different participants 
which constitute the criteria for making the judgments; 

- f i nal ly , the structure used to col 1 ect the i nf ormation emerged 
from and is transformed by the situations. 

The naturalistic approach must basically meet the following two 
conditions: 

1. the evaluator imposes a minimum cf constraints on the antecedents, 

2. the evaluator imposes a minimum of constraints on the parti- 
cipants 1 behaviors. 

Several models gather together under the general heading "Natu- 
ralistic approach". Among these we find the judiciary model (Owens, 
1973; Levine, 1974; Wolf, 1975); the transactional model (Rippey, 
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1973); the illuminating model (Parlett and Hamilton, 1976); the con- 
naisseur model (Eisner, 1976) and the responsive model (Stake, 1976; 
Guba and Lincoln, 1981; Borich and Jemelka, 1982). 
The responsive model 

The responsive model stresses most particularly preoccupations, 
questions, and problems encountered by the different audiences con- 
cerned by the program under evaluation. Therefore, important for the 
evaluator to have a pluralistic view of the program including the 
different points of view, and the emerging conflicts. In that sense, 
the responsive model structure is quite flexible and the evaluation 
process is likely to change on the basis of the in-coming infor- 
mation. Qualitative as well as quantitative analysis are acceptable. 
Most important, the evriuator is seen as a participant. 

According to Scriven (1978), Guba and Lincoln (1981), the 
responsive model of evaluation may take two major orientations [which 
complement each other (Guba and Lincoln, 1981)], namely: 

1. the detenni nation of the merit of the entity being eval uated 
or 

2. the determination of the value of the entity being evaluated. 

The merit is more an estimation of the absolute intrinsic 
quality whereas the value is a contextual related quality. The 
distinction between the two concepts is important since the judgments 
wil be different depending upon the chosen orientation. Tho assess- 
ment of the internal quality is often referred to as formative 
evaluation. 
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On the other hand, the assessment of the contextual quality of a pro- 
gram is referred to as summatWe evaluation. For Guba and Lincoln 
(1981) these dimensions complement each other. 

Insert table 1 here 



The evaluation study 

The program 

The program being evaluated was a Certificate in Industrial 
Rel at ions which is offered in a continuous education program in a 
Quebec French university. This program, of i proficiency type, was 
implemented in 1970. The expected students would come from theindus- 
trial relations field, or be at an executive or managerial level. 
The admissions requirements were: a first university degree or, a 
college diploma plus one year experience in Industrial Relations 
(I.R.) or, three years experience in I.R. Over 1800 students were 
enrolled from 1970 to 1982. 

In a previous study, Hurteau (1982) found some interesting pro- 
blems which convinced the administrators that something was wrong 
with the program and therefore prompted the evaluation process. 
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Among these we find: 

1. clients were different than those expected: in fact, 86% of them 
did not have any experience in I.R.; only 14% of the students were 
working in the field of I.R.; only 21% of the students had a first 
university degree whereas administrators expected almost all of them 
would so; 

2. an incredible rate of drop outs. In fact, over the years 52% of 
the students did drop out of the program; 

3. finally, the program had sustained no substantial changes since 
its implementation. 

On the basis of that information, the administrators accepted to 
have thei r program evaluated and , furthermore, they agreed to the 
idea of using a naturalistic approach. 

The evaluation process 
1. The local util ity 

Guba and Lincoln (1981) suggest the use of a need assessment to 
determine the local utility of a program (summative evaluation of the 
value). We did proceed as suggested by Kaufman and English (1979) 
and Witkin (1984), inducing potential needs (alpha needs) by inter- 
viewing workers in the field (administrators, labor union representa- 
tives, etc.), graduates of the program and lecturers in the program. 
A sample of 32 persons were asked the following question: "To what 
needs should the program answer in order to be useful to the students 
enrolled in the Industrial Relations Certificate?". The gathered 
information was submitted to a content analysis and the and result 
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contributed to the prel i mi nary version of the needs assessment 
instrument. The first draft questionnaire was then submitted to both 
admi ni strators and teaching staff who were requested to study, 
correct and complete it according to their experiment. This second 
step revealed itself unsuccessful. In order to elaborate the final 
version of the instru.nent, an analysis of the courses 1 objectives was 
done and interviews were conducted among the teaching staff. The 
resulting final needs assessment instrument (Likert scale) was 
administered to the 550 program's graduates who were asked two 
questions: 

1. n to what extent is the actual program, responding to each of the 
identified needs?" 

and 

2. "to what extent should it be responding to each of these needs?" 
The observed differences between the actual and the desired 

states of the 280 returned questionnai res (47% response rate) were 
analysed, and the local utility of the program determined. Among the 
findings, we found that 75% of the 33 potential needs were judged at 
least important and that the certificate as a whole did not seem, in 
its actual form, to offer an adequate coverage for them. 

2. The problems 

The information about the problems related to the program was 
collected in several steps: 
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f i rst, a seri e of informal intervi ew> was conducted among 
graduates of the program; they were asked to identify any problems 
related to the program whether academic, human, teaching staff, 
supervision, etc... and, 

second, a serie of interviews was conducted among drop outs 
(only 15 of them were located and accepted to participate) ana poten- 
tial drop outs of the program (a cample of 60 students who had not 
registered for two semesters in a row). They were asked two ques- 
tions , one about the reasons for dropi nq out of the program (or 
would make them drop out of the program) and the other, related to 
the problems associated with the program. 

A portrait of the situation was then sketched and submitted to 
the administrators who, at first, desagreed with the identified 
problems. A few months later, with their agreement and in order to 
validate the information, a questionnaire was sent to the entire 
student population. Results were similar to those of the interviews. 

The problems identified fell into four categories: students 1 
admission (laxism, students coming from different backgrounds, etc.); 
teach ing (absence of clear objectives , bad teach i ng, etc. ) and 
teaching staff (part time job to make some extra money, lack of 
interest in the program, etc.); students 1 supervision (no welcome 
group, offices closed because courses given at night; administrators 
non-available, ate); formation as a whole (overall evaluation 
positive but harsh comments about the program's administrators 
preoccupations, university's interest, etc.). 
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Advantages and limits of responsive evaluation 
A review of the literature about the responsive model of evalua- 
tion was conducted in order to list the advantages as well as dis- 
advantages relcted to the process. The idea was (1) to compare the 
findings in terms of similarities and differences and (2) to indicate 
any new thoughts derived from the present process and findings. 
Advantages 

Seme of the advantages listed in the literature were confirmed 
by the present study. 

1. As stated by Newcomer and Bernstein (1984); Stake and Hoke 
(1976); Stake and Pearsol (1981), the approach allows questions to 
emerge (the heterogeneity of the clientele, the absence of super- 
vision of the lecturers with respect to their syllabi, evaluation 
procedures, teaching, etc., the content overlap of several courses, 
etc., are examples of emerging information). This observation meets 
Rachkel's account (1976), who states that the approach allows the 
identification and the meeting of the clients 1 concerns. 
2*> We agree with Kalman (1 976), Rockwell (1982), Schermerhorn and 
Williams (1982), Preskill (1983), who all say that the data collected 
by this approach allow the evaluator to present a sketched portrait, 
accessible to conc?rned audiences and responding to their information 
needs. For instance, the content analysis of the information 
gathered by the interviews enabled us to identify some of the pro- 
blems faced by the program and to illustrate them with facts, textual 
notes . 
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3. As Scake (1983), we observed that immediate adjustments were 
possible when the gathered information was made available to t*.e 
decision-makers as soon as it emerged. For instance, in the present 
study, the fi rst present at i on or portrait of the p rob 1 ems provoked 
almost instant modifications. This observation is similar to the one 
made by Kalman (1976) anu Klintberg (1 976), for whom the results 
press to action and can easily be transformed into specific recomman- 
dations. 

4. We observed, as Schermerhorn and Williams (1082), Newcomer and 
Berstein (1984) did, that the evaluator 1 ^ immersion favours his rapid 
comprehension of the program. 

5. As Guba and Lincoln wrote (1981), with this approach, the values 
and the different points of view which influence the decisions are 
put forward and the direction of action emerge. 

Due to different ci rcumsta >ces , the following items were not 
sustained by the present study. 

1. Constant feedback between the eval uator and the stakehol ders 
does facilitate the acceptation of results (Schermerhorn and 
Williams, 1979; Newcomer and Berstein, 1984). 

2. Constant faedback between the eval uator and the stakeholders 
creates an atmosphere cf cooperation and sympathy toward the evalua- 
tion process (Rockwell, 1982; Michael, 1984; Newcomer and Berstein, 
1984). 

However, in previous study we found that constant feedback did 
in fact have the sDove advantages and it was one of the major reasons 
for us to choose the responsive evaluation in the present study. 
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The following advantages, though emerging from the study, are 
not mentioned in the literature. 

1. The responsive evaluation al lows the identification of resi s- 
tances and oppositions to the appro* -h or to the gathered infor- 
mations and the possibility to make needed adjustments. For ins- 
tance, following the presentation of the portrait of the situation 
sketched by means of interviews, the administrators showed strong 
desagreement, blamed the quality of the informants, questioned the 
credibility of the students... Letting things cooled down for a 
while, the evaluator suggested e cross validation of the information 
by means of a questionnaire to be sent to all students in the 
program. Being con 'inced of the importance of the information in 
thei r management process, the administrators agreed and actively 
participated in the process of building the questionnaire. 

2. The meaty description of the program and of its problems favors 
a certain transferability of information to similar ones. For 
instance, the gathered informatio.i about the problems of the Certifi- 
cate in Industrial Relations helped solving similar problems in other 
certificates. To some extent it helped administrators of other 
programs in their decision-making. 

Limits 

Some of the limits encountered in the present evaluation match 
those found in other studies: 

1. The time necessary to build the evaluation instruments can be 
lengthy (Klintberg, 1976). In the present study, the interviews, 
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their content analysis, the analyses of the courses 1 syllabi, the 
needs assessment questionnaire... were time consuming. 

2. The responsive approach implies a participation of the audiences 
to the construction and to the use of the instruments. According to 
Stake (1983), the persons involved do not always have the experience 
nor the expertise in evaluation and tnerefore probl ems may arise. 
The construction of the needs assessment questionnaire constitutes an 
example of such a problem; we had to change the initial procedures 
and the final version was not easy to come up with. 

3. The operating costs are quite high. Schermerhorn and Williams 
(1979) have compared the costs associated with both tKe naturalistic 
and the formalist models and found the ratio to be $6 ,000/$570.00. 
We can not substantiate those figures since we did not post any 
cost. However, it is our belief that we have done so the c^s would 
have been much higher using a responsive model. 

The literature identifies some limits that were not noticeable 
in our eva^at" 1 '' study: 

1. Accc .o Rackel (1976), Sorlie and Essex (1978), the 
approach is not always suitable to collect the necessary information 
and therefore the use of other models is sometime necessary to fill 
the limits of the responsive model. Rackel (1976) reserves the use 
of the responsive model to the evaluation of new programs. 

2. Van Hoose (1977) insists on the fact that, because of its flexi- 
bility, the responsive model can generate some difficulties in main- 
taining the focus and in collecting the information. 
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3. The results may seem oversimplified and disappointing, if one 
consider the way they are presented (portrait) (Stake and Pearson, 
1981). 

4. The approach is not so effective if one tries to find answers to 
specific questions, identified from the start or if one tries to 
prove rather than to comprehend (Stake, 1983). 

However, we have identified some disadvantages not found in the 
1 itprature. 

1. If the approach creates an atmosphere of cooperation and allows 
the identification of resistances, it can also keep the evaluator 
under the stakeholders 1 moods. For instance, when we presented to 
the administrators the portrait of the situation, their attitude was 
one of distrust. 

2* The responsive model allows the information to emerge and 
insists on the implication of the stakeholders. But, in doing so, 
the process can be slowed down. For instance, the interviews conduc- 
ted among the drop outs and graduates were useful in sketching the 
situation, but they also provoked harsh reactions from the adminis- 
trators and resulted in an important delay. 

3. The approach is very demanding on the evaluator and as such is 
not easily accessible. The evaluator must have a great capability of 
adaptation and lots of personal resources (managing the stakeholders 1 
resistances, adapting the information to the different audiences, 
etc.) as well as professional skills (conducting interviews, cons- 
tructing the evaluation instruments: needs assessment, question- 
naires, data analysis, etc.). 
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Table 1: Relationship of Merit/Worth to the Formati ve/Summati ve 
Distinction: A Curriculum Example 



Evaluation seeks to establish 



Type of Merit Worth 

Evaluation (Developmental Evaluation) (Adoptive Evaluation) 



Formative Intent: modify and improve 

design. 



Intent: fit entity to 
local context. 



Audience: entity develop- 
ment team. 



Audi ence : 1 ocal adapta- 
tion team. 



Source of Standards: panel 
of substantive experts. 



Source of Standards: 
assessment of local 
context and values. 



Summative Intent: critique, certify, 

and warrant entity. 



Intent: certify and 
warrant entity for 
local use. 



Audience: professional 
peers; potential 
adopters. 



Audience: local decision 
makers. 



Source of Standards: panel 
of substantive experts. 



Source of Standards: 
local needs assess- 
ment. 
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